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Abstract 


This  report  describes  the  development  of  a  new  screening  tool  for  identifying  Sailors 
with  high  potential  for  success  in  recruiting  duty.  In  the  first  phase  of  the  research 
effort,  a  predictive  validation  study  was  conducted  with  students  at  the  Recruiting  and 
Retention  School  in  Pensacola,  FL.  The  students  were  administered  a  trial  predictor 
battery  while  at  the  school,  and  performance  ratings  and  production  data  were  collected 
after  participants  had  been  assigned  to  recruiting  duty.  In  the  second  phase,  the  trial 
battery  was  revised  based  on  the  results  of  the  predictive  study,  and  validated  against 
performance  ratings  and  production  of  experienced  field  recruiters.  Suggestions  were 
made  regarding  a  final  scoring  key  and  how  the  test  battery  might  be  used  to  select  Navy 
personnel  for  recruiting  duty  and  possibly  other  special  assignment  ratings. 


I  ntroduction 


Navy  recruiters  face  an  increasingly  difficult  recruiting  environment  characterized  by 
lower  youth  enlistment  propensity  and,  during  the  1990s,  a  very  tight  labor  market. 

Over  the  last  several  years,  these  conditions  have  led  the  Navy  to  increase  its  recruiter 
force  by  approximately  500  recruiters  to  more  than  5,000.  In  order  to  staff  the  recruiter 
force,  E-5S  and  E-6s  must  be  moved  from  some  critically  undermanned  jobs  into 
recruiting  duty.  Because  recruiters  serve  as  representatives  of  the  Navy  to  the 
community  and  provide  the  critical  manpower  supply,  it  is  important  for  the  Navy  to 
make  good  decisions  about  which  of  these  E-5/6S  will  be  effective  as  recruiters. 

In  response  to  the  critical  recruiting  situation,  the  Navy  has  sponsored  Navy 
Personnel  Research,  Studies,  and  Technology  (NPRST)  to  conduct  a  research  and 
development  program  to  develop  recruiter  selection  and  assessment  systems  to  identify 
sailors  who  will  be  effective  and  productive  in  the  recruiting  environment.  Personnel 
Decisions  Research  Institutes,  Inc.  (PDRI)  was  contracted  by  NPRST  to  conduct  a  study 
to  evaluate  the  empirical  validity  of  a  paper-and-pencil  predictor  battery. 

The  purpose  of  the  current  project  was  to  develop  and  validate  a  test  battery  for  the 
selection  of  Navy  recruiters.  The  project  was  carried  out  in  two  phases.  The  first  phase 
involved  the  validation  of  several  instruments  thought  to  be  associated  with  recruiter 
performance. 

The  test  battery  in  the  first  phase  included  relevant  items  from  the  "Special 
Assignment  Battery"  (SAB)  developed  in  the  late  1970s  and  early  to  mid-1980s  by 
PDRI's  Borman  and  NPRDC's  Abrahams  (Atwater,  Abrahams,  &  Trent,  1986; 

Abrahams,  Neuman,  &  Rimland,  1973;  Borman,  Rosse,  &Toquam,  1981;  Borman, 

Rosse,  &  Toquam,  1979)  for  the  selection  of  Navy  and  Marine  Corps  recruiters.  The 
original  SAB  consisted  of  personality,  biographical,  and  vocational  interest  items.  In  the 
current  research,  a  five-factor  personality  measure  and  a  measure  of  emotional 
intelligence  were  also  included  in  the  battery. 
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The  second  phase  of  the  project  involved  revising  the  initial  test  battery  based  on  the 
results  from  the  first  phase  of  the  project  and  validating  the  revised  battery  in  a  different 
sample  of  recruiters,  using  a  concurrent  validation  design. 


Procedure/  Approach 


The  validation  approach  chosen  for  this  project  was  a  criterion-related  strategy.  This 
is  typically  accomplished  by  obtaining  the  test  scores  of  job  applicants  and  collecting 
measures  of  these  same  individuals’ job  performance.  Test  scores  are  then  related  to 
how  well  individuals  perform  on  the  job.  Successful  validation  of  this  type  provides 
confirmation  that  use  of  the  selection  measures  will,  in  fact,  identify  the  most  qualified 
candidates  for  the  job.  This  validation  methodology  is  one  of  three  validation  strategies 
presented  in  the  Uniform  Guidelines  on  Employee  Selection  Procedures  (1978,  EEOC), 
the  Standards  for  Educational  and  Psychological  Testing  (19),  and  the  Society  for 
Industrial/  Organizational  Psychology’s  (SIOP’s)  Principles  for  the  Validation  and  Use 
of  Personnel  Selection  Procedures  (1987).  A  predictive  strategy  was  used  during  the  first 
phase  of  the  project,  whereas  a  concurrent  strategy  was  employed  during  the  second 
phase.  The  specifics  of  the  data  collection  efforts  including  details  of  criterion 
development,  predictor  selection,  and  validation  of  the  predictors  are  provided  in  the 
following  sections. 


Phase  I  —Predictive  Validation  Study 


Development  of  Criterion  Measures  of  Recruiter  J  ob  Performance 

In  a  validation  study,  accurate  performance  measurement  is  critical  for  several 
reasons.  First,  in  order  to  demonstrate  the  utility  of  the  tests  for  selecting  qualified 
candidates,  it  is  necessary  to  show  that  they  are,  in  fact,  related  to  job  performance.  This 
can  only  be  achieved  if  comprehensive,  reliable,  and  valid  measures  of  job  performance 
are  available. 

The  goal  was  to  develop  criteria  that  would  fully  capture  the  Navy  recruiter  job 
performance  domain.  Specifically,  the  intent  was  to  develop  a  set  of  criterion  measures, 
each  with  its  own  strengths,  for  measuring  recruiter  performance.  Two  measures  of 
recruiter  job  performance  were  used  in  the  validation  study:  recruiter  rating  scales 
designed  to  measure  typical  performance  over  time,  i.e.,  what  each  recruiter  actually 
does  to  perform  effectively  or  ineffectively;  and  recruiter  production,  measuring  a 
critical  outcome  of  job  performance.  Each  criterion  measure  is  described  below. 
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Navy  Recruiter  Performance  Rating  Scales 

As  mentioned,  performance  ratings  scales  were  designed  to  measure  job 
performance  of  Navy  recruiters  across  all  behavioral  performance  requirements.  Two 
previous  reports  describe  the  original  development  of  these  behavior-based  scales 
(Borman,  Hough,  &  Dunnette,  1976)  and  the  recent  update  of  the  original  scales 
(Penney,  Borman,  Hedge,  Abrahams,  &  Drenth,  2001).  The  scales  feature  behavioral 
“anchors,”  providing  a  behavioral  description  of  how  individuals  at  different  levels  of 
effectiveness  perform  on  the  job.  This  method  has  been  shown  to  generate  more 
objective  and  reliable  performance  ratings.  The  eight  behavioral  dimensions  were:  (1) 
Locating  and  Contacting  Qualified  Prospects;  (2)  Gaining  and  Maintaining  Rapport;  (3) 
Obtaining  Information  from  Prospects  and  Making  Good  Person-Navy  Fits;  (4) 
Salesmanship  Skills;  (5)  Establishing  and  Maintaining  Good  Relationships  in  the 
Community;  (6)  Providing  Knowledgeable  and  Accurate  Information  about  the  Navy; 

(7)  Administrative  Skills;  and  (8)  Supporting  Other  Recruiters  and  the  Command. 

The  first  four  dimensions  clearly  represent  the  major  steps  that  recruiters  perform  in 
the  selling  process  (i.e.,  prospecting  for  candidates,  establishing  rapport,  making 
appropriate  judgments  about  the  fit  between  the  prospect  and  the  Navy,  and  closing  the 
sale).  Recruiters  must  also  initiate,  develop,  and  maintain  productive  relationships  with 
individuals  and  agencies  in  the  community  in  order  to  build  and  enhance  the  Navy's 
reputation  (Dimension  5).  Providing  accurate  information  about  the  Navy  (Dimension 
6),  planning,  organizing,  and  time  management  skills  referring  to  the  recruiter’s  ability 
to  balance  priorities  and  deadlines,  and  to  manage  enlistment  processing  (Dimension  7), 
and  coordinating  with  and  supporting  other  recruiters,  following  orders,  and  helping  or 
mentoring  other  recruiters  (Dimension  8)  are  all  additional  activities  important  for 
success  in  this  job. 

These  behavior-based  rating  scales  were  designed  to  encourage  raters  to  make 
evaluations  as  objectively  as  possible.  Specifically,  within  each  performance  dimension, 
statements  describing  behaviors  that  reflect  very  effective,  effective,  needs 
improvement,  and  needs  considerable  improvement  performance  anchor  these  four 
effectiveness  levels  on  the  scales.  Raters  are  asked  to  compare  observed  recruiter 
behavior  with  the  behavioral  statements  that  anchor  the  different  effectiveness  levels  on 
each  dimension.  The  Navy  Recruiter  Performance  Rating  Scales  appear  in  Appendix  A. 

In  addition,  a  rater  training  program  was  developed  to:  (1)  orient  raters  to  the  rating 
task;  (2)  familiarize  raters  with  the  performance  dimensions  and  how  each  is  defined; 

(3)  train  raters  to  match  observed  recruiter  behavior  with  the  behavioral  summary 
statements  to  obtain  a  rating  for  each  dimension;  (4)  describe  common  rater  errors 
(e.g.,  halo)  and  how  to  avoid  them;  and  (5)  encourage  raters  to  be  as  accurate  as  possible 
when  making  their  ratings.  The  rater  training  program  was  conducted  in-person  by  a 
PDRI  staff  member  during  the  rating  data  collection.  A  videotape  of  the  rater  training 
program  was  also  prepared  and  used  in  one  rating  data  collection  site. 
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Performance  Rating  Data  Collection 


We  developed  a  plan  to  collect  performance  ratings  from  both  peers  and  supervisors 
(Recruiters-in-Charge:  RINCs)  of  recruiters.  However,  discussions  with  NPRST  and 
CNRC  indicated  that  peers  and  RINCs  would  be  unavailable  to  provide  ratings. 
Therefore,  only  supervisory  ratings  from  Zone  Supervisors  were  collected.  Although 
Zone  Supervisors  may  not  have  as  much  daily  interaction  with  recruiters  as  RINCs,  our 
discussions  with  Zone  Supervisors  indicated  that  they  do  monitor  individual  recruiter 
performance  and  should  be  able  to  provide  accurate  ratings. 

The  plan  was  to  send  PDRI  staff  to  train  the  raters  and  collect  ratings  at  a  single 
location  within  each  District.  This  plan  was  carried  out  in  all  but  one  District.  There  the 
point  of  contact  received  the  ratings  materials  and  instructions  for  conducting  the 
ratings  sessions,  showed  the  training  video-tape  for  the  participating  raters,  collected 
the  performance  ratings,  and  mailed  the  ratings  back  to  PDRI.  Table  l  shows  the 
number  of  ratings  collected  and  the  method  used  at  each  site.  Performance  ratings  for 
134  recruiters  were  provided  by  41  supervisor  raters. 


Table  1 

Rating  scale  data  collection  by  site 


District 

Ratees 

Raters 

Criterion 

collection 

Atlanta,  GA 

30 

9 

1  n-person 

Dallas,  TX 

31 

7 

1  n-person 

Los  Angeles,  CA 

33 

9 

Video  mail-out 

Nashville,  TN 

17 

9 

In-person 

Richmond,  VA 

23 

7 

1  n-person 

Total 

134 

41 

In  addition,  we  asked  raters  how  long  they  had  worked  with  the  recruiter(s)  they 
were  evaluating.  The  mean  number  of  months  raters  had  worked  with  recruiters  was 
5.75.  Although  this  is  a  fairly  short  period  of  time,  it  is  not  surprising  given  that  the 
recruiters  in  the  sample  began  recruiter  training  at  the  Navy  Orientation  Recruiting  Unit 
(NORU)  between  July  2001  and  November  2001.  Interviews  with  recruiters  indicated 
that  raters  who  had  supervised  recruiters  for  less  than  two  months  likely  had 
insufficient  time  to  observe  and  accurately  evaluate  their  performance  and  these  rater  x 
ratee  combinations  were  removed  from  further  analyses.  This  resulted  in  only  10  fewer 
ratings  included  in  the  analyses  (N  =  124). 

Tables  2  and  3  illustrate  the  distribution  of  supervisor  ratings  across  the  10-point 
rating  scale.  There  is  a  low,  but  noteworthy  percentage  of  ratings  at  the  lower,  ineffective 
end  of  the  scale.  Most  of  the  ratings  fall  in  the  5-8  range,  and  there  is  reasonable 
variability  in  the  ratings,  suggesting  that  supervisors  were  not  inflating  their  ratings  and 
were  differentiating  between  the  more  and  less  effective  recruiters.  Table  4  shows  the 
means  and  standard  deviations  of  rating  scores  across  recruiters  for  each  dimension. 
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Table  2 

Number  and  percentage  of  supervisor  ratings  at  each  scale  point 


Rating  Scale  Point 
(1= Lowest, 

10=  Highest) 

Number  of  Ratings3 

Percentage  of  Ratings 

1 

6 

0.54 

2 

37 

3.32 

3 

73 

6.54 

4 

114 

10.22 

5 

186 

16.67 

6 

223 

19.98 

7 

231 

20.70 

8 

135 

12.10 

9 

89 

7.97 

10 

22 

1.97 

aTotal  number  of  supervisor  ratings  across  all  eight  dimensions  and  the  overall  performance  dimension. 


Table  3 

Mean  and  standard  deviations  for  mean  ratings  on  each  dimension 


Rating  Dimension  (N  =  124) 

Mean 

Standard 

Deviation 

Locating  and  Contacting  Qualified  Prospects 

5.38 

1.90 

Gaining  and  Maintaining  Rapport 

6.37 

1.95 

Obtaining  Information  from  Prospects 

5.55 

1.67 

Salesmanship  Skills 

5.37 

1.84 

Establishing  and  Maintaining  Good  Relationships  in 
the  Community 

6.62 

1.81 

Providing  Knowledgeable  and  Accurate  Information 
about  the  Navy 

6.38 

1.64 

Administrative  Skills 

5.79 

1.96 

Supporting  other  Recruiters  and  the  Command 

6.72 

1.92 

Overall  Performance 

6.18 

1.71 
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Table  4 

Mean  correlations  of  dimensions  within  and  between  factors 


Selling  Skills 
Factor 

Human 
Relations 
Skills  Factor 

Organizing 

Skills 

Factor 

Selling  Skills  Factor 

.60 

Human  Relations  Skills  Factor 

.44 

.53 

Organizing  Skills  Factor 

.35 

.42 

-na- 

Note:  The  Organizing  Skills  Factor  had  only  one  dimension  so  within  factor  mean  correlation  could  not  be 
computed. 


Factor  Analysis  of  the  Ratings 

To  examine  the  underlying  structure  of  the  eight  rating  scale  dimensions,  we 
conducted  a  principal  factors  analysis  with  a  varimax  rotation  on  the  supervisor  ratings. 
Results  of  this  analysis  suggested  that  a  one-factor  solution  provided  the  best 
description  of  the  data.  However,  work  we  have  done  in  the  past  with  Navy  and  Army 
recruiters  has  shown  that  a  three-factor  system  nicely  summarizes  the  recruiter 
performance  domain.  These  three  factors  are  Selling  Skills,  Human  Relations  Skills,  and 
Organizing  Skills. 

Past  research  has  shown  that  the  first  four  dimensions,  Locating  and  Contacting 
Qualified  Prospects,  Gaining  and  Maintaining  Rapport,  Obtaining  Information  from 
Prospects  and  Making  Good  Person-Army  Fits,  and  Salesmanship  Skills,  comprise  the 
Selling  Skills  Factor  l.  The  fifth  and  eighth  dimensions,  Establishing  and  Maintaining 
Good  Relationships  in  the  Community  and  Supporting  Other  Recruiters  and  the 
Command,  have  loaded  on  the  Human  Relations  Skills  factor.  The  seventh  dimension, 
Administrative  Skills,  has  loaded  its  own  factor  and  the  sixth  dimension,  Providing 
Knowledgeable  and  Accurate  Information  about  The  Navy,  has  not  loaded  cleanly  on 
any  of  the  three  factors. 

Although  the  initial  factor  analysis  that  was  conducted  did  not  support  this  three 
factor  solution,  we  decided  to  examine  the  rating  data  for  evidence  of  convergent  and 
discriminant  validity  for  the  three  factors.  Specifically,  we  computed  the  mean 
correlations  of  ratings  on  dimensions  within  the  hypothesized  three  factors  versus  the 
mean  correlations  between  dimension  ratings  on  different  factors.  The  analysis 
provided  some  indication  of  convergent  and  discriminant  validity  (see  Table  4).  That  is, 
we  found  that  the  correlations  for  dimensions  within  a  factor  were  higher  than  the 
correlations  for  dimensions  between  factors. 
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Based  on  this  analysis  and  because  the  three  rating  factors  allow  for  a  more  detailed 
interpretation  of  the  data  than  a  single  overall  performance  factor,  we  developed  three 
rating  scale  composite  scores  for  each  recruiter.  The  composites  were  computed  by  unit¬ 
weighting  the  standardized  scores  for  ratings  on  each  of  the  dimensions  loading  on  that 
factor.  Thus,  the  performance  ratings  capture  typical  recruiter  performance  over  time  on 
three  factors  that  appear  to  represent  in  summary  form  all  of  the  important 
performance  requirements  of  the  recruiter  job. 


Recruiter  Production 

Typically,  production  measures  focus  on  the  number  of  recruits  or  contracts  signed 
in  a  specified  time  period,  for  example,  number  of  recruits  signed  per  month. 

Production  measures  are  inherently  appealing  as  indicators  of  recruiter  performance. 
Not  only  are  they  "bottom-line"  measures,  but  also  they  are  apparently  objective,  and 
they  are  certainly  quantifiable  and  readily  available.  Unfortunately,  their  usefulness  as  a 
recruiter  performance  criterion  is  limited  in  several  ways. 

First,  a  recruiter's  production  may  be  influenced  by  territorial  or  situational  factors 
that  are  beyond  the  recruiter's  control.  A  number  of  researchers  have  attempted  to 
identify  and  examine  the  effects  of  territorial  factors  that  influence  production.  In  fact, 
Borman,  Rosse,  and  Toquam  (1982)  reviewed  many  of  these  studies.  The  primary 
finding  was  that  territorial  influences  can  be  shown  to  account  for  a  considerable 
amount  of  variance  in  recruiter  production.  Thus,  one  conclusion  to  be  drawn  from  this 
review  is  that  we  might  adjust  the  individual  production  scores  of  recruiters  according  to 
“scores”  for  a  territory  on  such  environmental  variables  as  propensity  for  youth  to  enlist, 
military-civilian  pay  ratio,  and  proportion  of  the  population  that  is  military,  or  retired 
military. 

However,  we  have  argued  (Borman  et  al.,  1982;  Borman  et  al.,  2001)  that  this 
strategy  for  correcting  production  data  according  to  territorial  influences  is  inferior  to 
simply  correcting  these  data  using  the  mean  production  levels  for  the  target  territories 
(e.g.,  Districts).  This  is  because  employing  a  territorial  average  actually  ensures  that  all 
environmental  factors  leading  to  territorial  differences  are  indeed  being  accounted  for. 
That  is,  using  territorial  averages  to  make  these  corrections  captures  the  influence  of  all 
relevant  environmental  factors  on  production. 

A  second  issue  regarding  the  deficiency  of  raw  production  indices  and  the  notion  of 
correcting  production  scores  with  territorial  averages  is  the  size  of  the  territory  to 
employ  in  making  the  score  adjustments.  Using  relatively  large  territories  such  as 
Regions  has  the  advantage  of  providing  more  stable  means,  with  relatively  small  effects 
due  to  uneven  recruiter  talent  across  the  Regions.  However,  a  disadvantage  with 
Regions  is  that  the  recruiting  environment  may  vary  in  difficulty  across  different 
sections  of  the  Region,  making  a  single  adjustment  for  all  recruiters  in  the  Regions 
potentially  misleading.  The  advantages  and  disadvantages  of  a  smaller  territory,  such  as 
at  the  District  level,  are  the  opposite.  The  homogeneity  in  the  environmental  factors  is 
likely  to  be  high,  but  the  District  means  will  be  less  stable  and  more  highly  influenced  by 
uneven  recruiter  talent  in  different  Districts. 
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One  last  issue  with  adjusting  production  data  using  territorial  averages  is  how  long  a 
time  period  to  employ  when  computing  average  production  scores.  In  general,  longer 
(i.e.,  more  months)  is  better,  because  the  stability  of  the  means  is  likely  to  be  higher 
when  several  months'  data  are  averaged.  However,  because  recruiters  leave  active 
recruiting  status  and  new  recruiters  join  the  recruiting  force,  the  greater  the  number  of 
months  used  for  computation  of  means,  the  larger  the  missing  data  problem,  with 
recruiters  being  on  active  recruiting  duty  status  for  only  part  of  the  period. 

In  sum,  several  issues  must  be  considered  to  improve  the  usefulness  of  production 
averages  as  an  indicator  of  recruiter  effectiveness.  Addressing  each  of  these  concerns 
can  greatly  improve  the  utility  of  production  as  an  indicator  of  recruiter  effectiveness. 
The  current  study  took  these  factors  into  account  in  the  development  of  the  production 
index.  The  specific  steps  taken  are  described  in  the  next  section. 


Production  I  ndex  for  the  Predictive  Validation  Study 

CNRC  maintains  a  database  tracking  the  monthly  production  of  all  Navy  recruiters. 
For  the  first  phase  of  the  project,  we  obtained  monthly  information  from  July  2001 
through  June  2002  regarding  the  gross  number  of  recruits  signed.  Descriptive  statistics 
for  production  in  CNRC  and  the  test  sample  are  presented  in  Table  5. 


Table  5 

Average  monthly  uncorrected  (raw)  production:  descriptive  statistics 


N 

Minimum 

Maximum 

Mean 

Standard 

Deviation 

Validation  Sample 

572 

0.11 

3.00 

1.16 

0.58 

All  of  CNRC 

4863 

0.08 

5.17 

1.25 

0.74 

Although  we  obtained  data  for  12  months,  not  all  recruiters  were  "on-production"  for 
all  12  months.  That  is,  some  recruiters  did  not  have  production  data  for  all  12  months 
because  they:  (1)  began  active  recruiting  duty  after  July  2001;  (2)  took  temporary  leave 
during  the  12-month  period;  or  (3)  were  temporarily  or  permanently  removed  from 
active  recruiting  duty  for  some  other  reason.  On  average,  recruiters  in  our  validation 
sample  (N  =  572)  had  9.8  months  of  production  data. 

In  order  to  develop  an  appropriate  production  index,  the  issues  discussed  earlier  in 
the  report  were  addressed.  First,  we  took  steps  to  correct  for  situational  and 
environmental  factors  that,  as  we  mentioned  previously,  may  impact  recruiter 
production.  One  problem  that  can  affect  a  recruiter's  production  is  the  months  in  which 
they  are  recruiting.  For  example,  signing  four  individuals  in  June  may  not  require  the 
same  skill  as  signing  four  individuals  in  January  due  to  the  influx  of  high  school 
graduates  into  the  market  during  the  early  summer  months.  In  order  to  correct  for 
seasonal  differences  that  may  make  recruiting  in  one  month  more  difficult  than  another, 
production  values  were  first  standardized  within  each  month  before  they  were  combined 
to  form  a  monthly  standardized  average. 
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In  addition,  we  accounted  for  environmental  factors  employing  the  strategy 
discussed  previously  by  using  territorial  averages  to  correct  production  scores.  First,  we 
created  production  scores  for  each  recruiter  standardized  within  District  and  Region  in 
order  to  determine  the  level  that  would  be  most  appropriate.  Next,  we  examined  the 
reliabilities  of  each  standardized  production  score.  The  reliabilities  for  the  Region-  and 
District-corrected  scores  were  moderately  high  (r  =  .80  and  .79,  respectively)  for  an 
average  of  9.8  months.  Overall,  the  Region  -level  correction  seemed  to  be  preferable 
because  of  its  leading  to  slightly  higher  reliability. 

The  last  issue  addressed  regarding  the  production  data  was  the  number  of  months  to 
employ  in  computing  the  averages.  As  stated  previously,  on  average,  we  had  9.8  months 
of  production  data  for  the  recruiters  in  our  sample.  Thus,  the  final  production  score  for 
some  recruiters  was  based  on  as  few  as  3-4  months  of  data,  whereas  others  had  as  many 
as  12  months.  Because  the  stability  of  the  production  score  is  likely  to  be  higher  when 
more  months'  data  are  averaged,  including  scores  based  on  only  a  few  months  of  data 
may  attenuate  the  observed  relationships  with  the  predictors,  as  well  as  other  criteria. 
We  therefore  examined  the  reliability  of  production  scores  based  on  varying  number  of 
months'  data  (see  Table  6)  to  determine  an  appropriate  cut-off.  As  expected,  the 
reliability  of  production  scores  increases  as  the  number  of  months  averaged  increases.  A 
reasonable  cut-off  appears  to  be  at  four  months  or  more  data.  The  reliability  of  the 
production  data  drops  fairly  substantially  with  fewer  months  data.  Based  on  this  finding 
and  the  number  of  recruiters  with  different  numbers  of  months  data  (e.g.,  many  more 
recruiters  had  4  or  5  months  data  compared  to  11  or  12  months  data),  we  decided  to 
screen  out  those  individuals  with  less  than  four  months  of  production  data. 


Table  6 

Reliabilities  of  production  indices  using  different  time  intervals 


Time  Length 

Reliability 

12  months 

.80 

11  months 

.80 

10  months 

.80 

9  months 

.80 

8  months 

.79 

7  months 

.78 

6  months 

.77 

5  months 

.75 

4  months 

.72 

3  months 

.68 

2  months 

.62 
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Correlations  Between  the  Criterion  Measures 

Table  7  depicts  the  relationships  between  the  three  performance  rating  factors  and 
the  Region-adjusted  production  index.  Production  correlates  highest  with  the  Selling 
Skills  rating  factor  (.61),  the  factor  that  most  closely  represents  the  skill,  effort,  and 
ability  of  recruiters  to  enlist  prospects.  Production  correlates  less  highly  with  the 
Human  Relations  Skills  (.33)  and  Organizing  Skills  (.24)  rating  factors.  These  two 
factors  represent  activities  that  are  more  likely  to  have  a  long-term  impact  on  recruiter 
performance  rather  than  having  an  immediate  impact  on  monthly  production. 
Accordingly,  this  pattern  of  relations  provides  construct  validity  evidence  for  both  the 
ratings  and  production  index. 


Table  7 

Correlations  between  criterion  measures 


Criterion  Measure 
(N=123) 

Production 

Selling 

Skills 

Human 

Relations 

Skills 

Organizing 

Skills 

Overall 

Performance 

Selling  Skills 

.61** 

Human  Relations  Skills 

.33** 

.59** 

Organizing  Skills 

.23* 

40** 

4g** 

*<  .05 
**  p  <  .01 


Predictor  Measures  and  Data  Collection 

Four  predictor  instruments  measuring  personality,  vocational  interests,  and 
emotional  intelligence  were  used  in  the  validation  study.  Descriptions  of  each  are 
provided  below. 


Special  Assignment  Battery  (SAB) 

As  stated  previously,  the  Special  Assignment  Battery  (SAB)  was  developed  for  the 
selection  of  Navy  and  Marine  Corps  recruiters  in  the  late  1970s  and  early  to  mid  1980s. 
The  SAB  consists  of  two  subtests.  The  first  subtest,  the  Work  Style  Inventory  (WSI),  is  a 
self-report  measure  of  personality.  The  WSI  contains  179  items  and  is  scored  on  5-point 
Likert  scale  ranging  from  "Definitely  false"  to  "Definitely  true."  The  five  personality 
traits  assessed  by  the  WSI  are  Ambition  (e.g.,  "I  often  set  higher  standards  for  myself 
than  others  set  for  themselves,"  "Most  people  would  agree  that  I  am  an  overachiever"), 
Conscientiousness  (e.g.,  “I  make  sure  my  daily  life  is  planned  and  organized,"  "I  obey  a 
rule  even  if  everybody  around  me  is  not  obeying  it"),  Emotional  Stability  (e.g.,  "It  seems 
like  I  am  always  trying  to  turn  my  life  around,"  "There  are  times  when  I  feel  I  am  no 
good  to  anyone,"  both  negatively  scored),  Extroversion  (e.g.,  “When  I  talk  to  people  they 
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get  energized,"  "I  find  it  easy  to  start  up  a  conversation  with  strangers"),  and  Leadership 
(e.g.,  “I  enjoy  persuading  others  to  accept  my  point  of  view,"  "I  enjoy  or  would  enjoy 
having  power  over  others"). 

The  second  subtest,  the  Vocational  Interest  Career  Examination  (VOICE),  assesses 
vocational  interests.  Respondents  are  presented  with  a  list  of  250  jobs,  work  tasks,  and 
leisure  activities  and  asked  to  indicate  whether  they  would  like,  dislike,  or  are  indifferent 
to  performing  or  learning  more  about  those  activities. 

NEO 

The  NEO  is  a  measure  of  the  five  major  domains  of  personality  (i.e.,  the  Big  5; 
Goldberg,  1993)  and  important  traits  or  facets  that  define  each  domain.  The  NEO  long- 
form  is  a  self-report,  240-item  measure.  The  NEO  assesses  the  facets  of  Neuroticism 
(adjustment  or  emotional  stability),  Extroversion  (sociability,  assertiveness),  Openness 
(active  imagination,  aesthetic  sensitivity,  attentiveness  to  inner  feelings),  Agreeableness 
(interpersonal  tendencies  such  as  altruism,  sympathy  to  others),  and  Conscientiousness 
(impulse  control,  and  planning  and  organizing).  The  scales  were  developed  and  refined 
by  a  combination  of  rational  and  factor  analytic  methods  and  have  been  the  subject  of 
intensive  research  conducted  over  15  years  on  both  clinical  and  normal  adult  samples. 
Several  of  the  Big  Five  dimensions  have  been  linked  with  sales  performance  (Barrick, 
Mount  &  Strauss,  1993;  Mount  &  Barrick,  1998;  Vinchur,  Schippmann,  Switzer,  &  Roth, 
1998),  and  other  research  has  provided  criterion-related  validity  evidence  for  the  NEO 
in  sales  positions  (e.g.,  Stewart,  1996). 

EQ-I 

The  BarOn  EQ-I  is  a  measure  of  emotional  intelligence.  Emotional  intelligence  is 
defined  as  an  array  of  non-cognitive  capabilities,  competencies,  and  skills  that  influence 
one’s  ability  to  succeed  in  coping  with  environmental  demands  and  pressures.  The 
BarOn  EQ-I  measures  15  conceptual  components  of  emotional  intelligence.  These  are: 
Emotional  Self-Awareness,  Assertiveness,  Self-Regard,  Self-Actualization, 

Independence,  Empathy,  Interpersonal  Relationship,  Social  Responsibility,  Problem 
Solving,  Reality  Testing,  Flexibility,  Stress  Tolerance,  Impulse  Control,  Happiness,  and 
Optimism.  The  measure  consists  of  133  items. 

Predictor  Data  Collection 

Predictor  test  data  were  collected  from  623  students  during  their  first  week  of 
recruiter  training  at  the  Navy  Orientation  Recruiting  Unit  (NORU)  in  Pensacola,  FL. 

Test  packets  were  assembled  with  the  predictor  instruments,  answer  sheets,  and 
instructions,  and  were  mailed  to  NORU.  The  packages  included:  (1)  a  participant 
information  sheet  (i.e.,  demographic  information);  (2)  the  test  instruments;  and  (3) 
answer  sheets.  The  participant  information  sheet  can  be  found  in  Appendix  B. 
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One  individual  at  NORU  was  assigned  to  proctor  the  predictor  test  administration. 
This  person  was  responsible  for  assembling  the  test  packets,  overseeing  the 
administration  of  the  test  battery,  and  mailing  the  completed  answer  sheets  to  PDRI. 

The  proctor  was  given  a  script  to  be  read  at  the  beginning  of  each  administration  in 
order  to  standardize  the  testing  process.  The  proctor  script  is  in  Appendix  C.  At  the  end 
of  August  2001,  administration  of  the  EQ-I  to  students  at  NORU  was  ceased  at  the 
request  of  NORU  officials  in  order  to  reduce  the  total  length  of  the  battery.  As  a  result, 
the  EQ-I  was  only  administered  to  199  students  and  the  number  of  students  for  whom 
rating  data  were  available  was  only  33.  Consequently,  the  sample  size  for  the  EQ-I  was 
too  low  to  complete  criterion-related  validation  analyses. 

Data  Screening 

After  receiving  the  data,  PDRI  screened  them  for  inconsistent  or  careless  responses. 
The  WSI  contains  two  items  designed  to  detect  invalid  responses.  These  items 
instructed  respondents  to  choose  a  specific  response  option.  For  example,  "This  item  for 
keypunch  purposes  only.  Please  select  response  choice  "a"."  Using  these  items  to  screen 
for  invalid  responses,  36  participants  were  flagged  and  subsequently  removed  from  the 
database  and  were  not  included  in  any  analyses. 

Additionally,  we  visually  inspected  each  test  to  identity  respondents  who  appeared  to 
answer  in  a  careless  or  intentionally  distorted  fashion.  When  inspecting  each  test,  we 
flagged  those  that  met  one  of  the  following  criteria:  (1)  more  than  10  responses  were 
missing;  (2)  a  subject  provided  identical  responses  to  25  questions  or  more  in  a  row;  and 
(3)  the  pattern  of  responding  clearly  indicated  the  subject  was  not  reading  questions 
(e.g.,  zigzag  answer  pattern,  etc.). 

Only  three  of  the  623  cases  were  flagged  as  a  result  of  this  screening  process.  In  total, 
39  of  the  tests  (6%)  were  judged  to  have  invalid  response  patterns.  These  tests  were 
subsequently  removed  from  the  database  and  not  included  in  any  of  the  analyses. 

Predictor  Descriptive  Data 

After  the  data  were  screened,  they  were  entered  manually  or  electronically  scanned 
into  computerized  data  files.  All  data  were  combined  into  a  single  file  using  SPSS 
statistical  software.  Preliminary  descriptive  analyses  were  performed  (e.g.,  range,  mean, 
standard  deviation  analyses)  as  a  final  check  to  identity  data  entry  errors,  missing 
responses,  or  other  possible  evidence  of  respondent  response  sets.  Item  level  data  for 
the  NEO  and  WSI  were  used  to  create  scale  and  facet  scores.  The  descriptive  statistics 
for  the  scale  scores  are  presented  in  Tables  8,  9,  and  10.  The  VOICE  section  of  the  SAB 
did  not  have  scales.  Subsequent  analyses  were  performed  at  the  item  level. 
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Table  8 

NEO  descriptive  statistics 


Scale 

N 

Mean 

Standard 

Deviation 

Neuroticism 

583 

73.86 

19.86 

Extroversion 

583 

117.87 

17.86 

Openness 

583 

106.91 

13.41 

Agreeableness 

583 

111.42 

14.35 

Conscientiousness 

583 

124.36 

18.14 

Table  9 

WSI  descriptive  statistics 

Scale 

N 

Mean 

Standard 

Deviation 

Ambition 

584 

66.47 

7.41 

Conscientiousness 

584 

102.61 

12.90 

Extroversion 

584 

119.76 

17.09 

Leadership 

584 

46.91 

6.88 

Emotional  Stability 

584 

49.03 

9.81 

Table  10 

EQI  descriptive  statistics 

Scale 

N 

Mean 

Standard 

Deviation 

Assertiveness 

199 

102.28 

13.59 

Emotional  Self-Awareness 

199 

98.61 

13.70 

Empathy 

199 

86.85 

17.44 

Flexibility 

199 

99.93 

15.55 

Happiness 

199 

96.09 

17.78 

Impulse  Control 

199 

99.51 

13.99 

Independence 

199 

99.59 

15.31 

Interpersonal  Relationship 

199 

93.67 

15.51 

Negative  Impression 

199 

106.01 

22.56 
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Table  10 

EQI  descriptive  statistics 


Scale 

N 

Mean 

Standard 

Deviation 

Optimism 

199 

93.05 

14.97 

Positive  1  mpression 

199 

99.90 

18.23 

Problem  Solving 

199 

96.28 

14.68 

Reality  Testing 

199 

94.30 

17.20 

Self-Actualization 

199 

92.75 

15.98 

Self-Regard 

199 

98.56 

13.63 

Social  Responsibility 

199 

87.67 

18.07 

Stress  Tolerance 

199 

99.81 

14.11 

Total  Score 

199 

94.04 

17.63 

Construct  validity  of  the  five  WSI  factors  was  demonstrated  by  their  correlations 
with  the  five  NEO  factors  (see  Table  li).  The  SAB  Emotional  Stability  factor  was 
correlated  -.55  with  the  NEO  Neuroticism  factor.  The  SAB  and  NEO  Extroversion 
factors  and  Conscientiousness  factors  were  correlated  .57  and  .53,  respectively. 

Table  11 

Correlations  between  NEO  and  WSI  factors 

NEO  Factors 

WSI  Factors  Neuroticism 

Extroversion 

Openness  Agreeableness 

Conscientiousness 

Ambition  -.16** 

28** 

.03  .03 

.38** 

Conscientiousness  -.14* 

.18** 

-.09*  .31** 

.53** 

Extroversion  -.27** 

.57** 

.22**  .05 

.23** 

Leadership  -.07 

28** 

16**  -.32** 

.03 

Emotional  Stability  -.55** 

.41** 

.20**  16** 

.43** 

N=583 
*  p  <  .05 
**  p  <  .01 
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Validation  Results 

This  section  presents  the  observed  criterion-related  validities  for  the  NEO  and  WSI. 
Recall  that  the  EQ-I  was  not  included  in  these  analyses  because  of  low  Ns.  To  examine 
the  validity  of  these  measures,  we  correlated  the  scale  scores  obtained  from  each 
measure  with  two  criterion  measures:  (l)  the  composite  criterion  ratings  on  the  three 
unit  weighted  composites;  and  (2)  production  (production  corrected  within  Region). 
Results  of  this  analysis  are  presented  in  Tables  12  and  13. 


Table  12 

Correlations  between  WSI  factors  and  criteria 


Selling 

Skills 

Human 

Relations 

Skills 

Organizing 

Skills 

Overall 

Performance 

Production 

Ambition 

.05 

.02 

-.09 

.07 

.10 

Conscientiousness 

-.02 

-.09 

.02 

-.04 

.07 

Extroversion 

* 

00 

1— 1 

.21* 

-.07 

.20* 

.13** 

Leadership 

.02 

.02 

-.13 

-.01 

.05 

Emotional  Stability 

.27** 

.26** 

.14 

.27** 

.10* 

N  =  119  for  Rating  Criteria 

N  =  535  for  Production 

Table  13 

Correlations  between  NEO  factors  and  criteria 


Human 

Selling  Relations  Organizing  Overall 


Skills 

Skills 

Skills 

Performance 

Production 

Neuroticism 

-.13 

-.21* 

OO 

O 

1 

-.14 

-.04 

Extroversion 

.16 

.14 

.05 

.14 

.09* 

Openness 

O 

O 

.06 

.09 

-.02 

-.01 

Agreeableness 

-.07 

.01 

.04 

OO 

O 

1 

.01 

Conscientiousness 

.09 

.11 

.10 

.03 

.04 

N  =  119  for  Rating  Criteria 

N  =  535  for  Production 
*  p  <  .05 
**  p  <  .05 
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In  general,  the  WSI  factors  were  more  valid  against  the  performance  rating  and 
production  criteria  than  the  NEO  factors.  The  WSI  Extroversion  factor  was  valid  against 
Selling  Skills  (.18),  Human  Relations  Skills  (.21),  Overall  Performance  (.20)  and 
production  (.13).  The  NEO  Extroversion  factor  was  only  valid  against  production  (.09). 
Similarly,  the  WSI  Emotional  Stability  factor  was  valid  against  Selling  Skills  (.27), 
Human  Relations  Skills  (.26),  Overall  Performance  (.27)  and  production  (.10).  The  NEO 
Neuroticism  factor  was  only  valid  against  Human  Relations  Skills  (-.21). 

As  mentioned,  an  item-level  empirical  validation  approach  was  used  to  evaluate  the 
VOICE.  There  were  a  total  of  250  VOICE  items  and  5  criteria,  and  thus  a  total  of  1,250 
validity  estimates.  The  validity  "hit-rate"  for  the  VOICE  items  at  the  .05  level  was  fairly 
low  (3.76%).  Only  47  items  had  a  significant  correlation  with  at  least  one  of  the  criteria. 
This  was  somewhat  surprising  given  that  a  meta-analysis  of  sales  performance 
predictors  (Vinchur  et  al.,  1998)  reported  that  vocational  interest  measures  are 
reasonably  valid  predictors  of  sales  performance.  Our  conclusion  was  that  the  VOICE 
was  not  a  valid  predictor  of  job  performance,  at  least  in  this  sample  of  recruiters. 


Phase  1 1  —Concurrent  Validation  Study 


Revising  the  Test  Battery 

The  long-term  goal  of  the  Navy's  work  in  recruiter  selection  is  to  create  a  valid 
selection  battery  that  can  be  easily  administered  in  a  relatively  short  amount  of  time. 
The  first  task  of  the  second  phase  of  the  project,  therefore,  was  to  revise  the  initial  test 
battery  to  maximize  criterion-related  validity  and  reduce  the  length  of  the  battery  so 
that  it  can  be  administered  in  approximately  30  minutes.  Several  factors  were  taken  into 
consideration  in  choosing  the  items  for  a  revised  battery.  First,  we  considered  the 
validity  of  each  individual  instrument  that  comprised  the  selection  battery.  Next,  we 
examined  the  validities  of  individual  items  and  investigated  the  factor  structure  of  the 
valid  items.  Finally,  additional  items  from  the  battery  were  chosen  based  on  content 
similarity  to  the  valid  factors. 

As  noted  previously,  of  the  two  personality  measures  in  the  battery,  the  WSI  factors 
were  more  valid  against  the  rating  and  production  criteria  than  the  NEO  factors. 

Because  of  the  higher  validities  achieved  with  the  WSI  personality  factors,  the  NEO  was 
excluded  from  the  revised  battery.  Additionally,  because  the  item-level  validity  hit-rate 
for  the  VOICE  items  was  quite  low,  the  VOICE  was  also  eliminated  from  the  revised 
battery.  Thus,  the  WSI  became  the  focus  of  the  revised  battery. 

A  two-step  approach  was  used  to  identify  WSI  items  for  inclusion  in  the  revised 
battery.  First,  items  were  selected  based  on  their  validity  against  the  performance  rating 
and  production  criteria.  Sixty-seven  items  were  identified  with  a  validity  above  .08 
against  production  or  above  .14  against  the  rating  criteria  and  were  included  in  the 
revised  battery.  In  addition  to  selecting  valid  items,  other  WSI  items  were  examined  for 
inclusion  in  the  battery  based  on  content.  To  accomplish  this,  a  factor  analysis  was 
conducted  on  the  67  relatively  valid  items  to  obtain  a  summary  description  of  their 
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underlying  structure.  The  factor  analysis  yielded  four  factors  (Extroversion,  Emotional 
Stability,  Leadership /Ambition,  and  Conscientiousness).  The  inference  from  the  factor 
analysis  results  is  that  these  four  factors  reflect  the  underlying  content  of  the  valid 
items. 

Finally,  PDRI  staff  members  appraised  the  content  of  the  remaining  WSI  items 
(items  with  non-significant  correlations  with  the  criteria)  relative  to  the  content  of  the 
items  within  each  of  the  valid  factors.  Items  whose  content  was  judged  to  be  similar  to 
the  valid  factors  and  whose  validities  were  reasonably  close  to  significance  were 
included  in  the  revised  battery.  Based  on  these  criteria,  40  additional  items  were 
included,  along  with  two  items  to  detect  invalid  responses,  for  a  total  of  109  items.  The 
revised  battery,  named  the  Recruiter  Assessment  Battery  (RAB),  contains  five  scales:  (a) 
Ambition;  (b)  Conscientiousness;  (c)  Extroversion;  (d)  Leadership;  and  (e)  Emotional 
Stability.  A  description  of  the  scale  definitions  can  be  found  in  Appendix  D. 


Data  Collection 

In  order  to  obtain  a  large  enough  sample  of  current  recruiters  in  the  concurrent 
validation  effort,  an  email  was  sent  by  the  Assistant  Deputy  Chief  of  Naval  Operations 
(Manpower  and  Personnel),  Code  NiB,  to  all  Navy  Recruiting  Districts  (NRDs)  asking 
for  NRDs  to  volunteer  to  participate  in  the  second  phase  of  the  project  during  their 
annual  training  and  awards  banquets.  Three  NRDs  volunteered  to  participate  in  the 
study  (Montgomery,  Omaha,  and  San  Antonio). 

At  each  of  these  three  NRDs,  PDRI  and  NPRST  staff  trained  the  Zone  Supervisor, 
RINC,  and  recruiter  peer  raters,  collected  performance  rating  data,  and  administered 
the  RAB  to  all  recruiters.  All  participants,  including  recruiters,  RINCs,  and  Zone 
Supervisors,  were  given  a  personalized  set  of  materials  that  included:  (1)  a  participant 
information  sheet  (i.e.,  demographic  information);  (2)  an  informed  consent  form;  (3) 
the  recruiter  performance  rating  scales  with  instructions;  and  (4)  a  performance  rating 
sheet  with  names  and  code  numbers  of  the  individuals  they  would  be  rating.  In  order  to 
protect  the  confidentiality  of  the  ratings,  raters  were  instructed  to  tear  off  the  section  of 
the  rating  form  with  the  names  after  they  had  completed  their  ratings.  Copies  of  these 
materials  can  be  found  in  Appendices  E-G.  Additionally,  all  recruiters  were  given  the 
RAB  and  an  answer  sheet.  The  code  numbers  for  each  recruiter,  placed  on  the  rating 
form  and  the  RAB  rating  sheet,  allowed  us  to  match  test  and  rating  data. 


Criterion  Data 

Performance  Rating  Scale  Data 

We  attempted  to  collect  ratings  from  two  supervisors  and  at  least  two  peers  for  each 
recruiter  in  the  study.  Table  14  shows  the  number  of  ratings  collected  at  each  site. 
Performance  ratings  for  278  recruiters  were  provided  by  234  peer,  69  RINC,  and  21 
Zone  Supervisor  raters,  for  a  total  of  1087  rater-ratee  pairs.  Table  15  shows  the  number 
of  supervisor  and  peer  raters  for  each  recruiter. 
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Table  14 

Rating  scale  data  collection  by  site 


NRD 

Raters 

Ratees 

Montgomery 

122 

104 

Omaha 

66 

56 

San  Antonio 

136 

118 

Total 

324 

278 

Table  15 

Number  of  supervisor  and  peer  raters 

Number  of 

Supervisor  Raters 

per  Ratee  N 

Number  of 
Peer  Raters 
per  Ratee 

N 

Total  Number 
of  Raters  per 
Ratee 

N 

0 

6 

0 

42 

1 

17 

1 

64 

1 

68 

2 

45 

2 

201 

2 

57 

3 

60 

3 

7 

3 

61 

4 

51 

4 

1 

4 

30 

5 

68 

5 

19 

6 

14 

6 

1 

7 

21 

7 

1 

8 

2 

9 

1 

Mean  number  of  supervisor  raters  per  ratee  =  1.76 
Mean  number  of  peer  raters  per  ratee  =  2.13 
Mean  total  number  of  raters  per  ratee  =  3.90 


Only  raters  who  had  worked  with  recruiters  for  at  least  2  months  were  included  in 
the  analyses.  This  resulted  in  only  93  (8.6%)  fewer  rater-ratee  pairs.  The  mean  number 
of  months  raters  had  worked  with  recruiters  was  9.2  months  for  peer  raters  and  10.6 
months  for  supervisor  raters. 

Rating  scores  were  created  for  each  recruiter  by  computing  the  mean  rating,  over  all 
raters,  for  each  dimension.  Table  16  shows  the  means  and  standard  deviations  of  these 
combined  rating  scores  for  each  dimension. 
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Table  16 

Mean  and  standard  deviations  for  combined  ratings  on  each  dimension 


Rating  Dimension 

Mean3 

Standard 

Deviation 

Locating  and  Contacting  Qualified  Prospects 

6.24 

1.50 

Gaining  and  Maintaining  Rapport 

6.86 

1.39 

Obtaining  Information  from  Prospects  and 
Making  Good  Person-Navy  Fits 

6.42 

1.31 

Salesmanship  Skills 

6.37 

1.44 

Establishing  and  Maintaining  Good 

Relationships  in  the  Community 

6.46 

1.32 

Providing  Knowledgeable  and  Accurate 

1  nformation  about  the  Navy 

7.01 

1.17 

Administrative  Skills 

6.06 

1.45 

Supporting  Other  Recruiters  and  the 

Command 

6.83 

1.48 

Overall  Performance 

6.60 

1.42 

aN=  268 


Table  17  presents  the  interrater  reliabilities  for  the  supervisor  and  peer  ratings 
combined.  Both  rating  sources  should  provide  important  performance  information 
because  of  their  unique  perspectives,  and  the  reliabilities  for  both  sources  taken  together 
argues  for  the  use  of  an  aggregated  supervisor/peer  rating  criterion. 
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Table  17 

I  nterrater  reliabilities  for  combined  supervisor,  peer  ratings3 


Combined 

Rating  Dimension 

Peer/  Supervisor 
Reliabilities13 

Locating  and  Contacting  Qualified  Prospects 

.81 

Gaining  and  Maintaining  Rapport 

.69 

Obtaining  Information  from  Prospects  and  Making  Good 
Person-Navy  Fits 

.65 

Salesmanship  Skills 

.75 

Establishing  and  Maintaining  Good  Relationships  in  the 
Community 

.58 

Providing  Knowledgeable  and  Accurate  Information  About 
the  Navy 

.60 

Administrative  Skills 

.61 

Supporting  Other  Recruiters  and  the  Command 

.59 

Overall  Performance 

.80 

Reliabilities  are  intraclass  correlation  coefficients  (ICC  l,k)(Shrout  &  Fleiss,  1979). 
bN  =  231  k  =  3.90 


As  in  the  predictive  validation  study,  we  combined  the  rating  dimensions  into  a  3- 
category  structure  for  the  validation.  Table  18  presents  the  means,  standard  deviations, 
and  interrater  reliabilities  of  these  three  composites. 


Table  18 

Means,  standard  deviations,  and  interrater  reliabilities  for  rating 

composites 


Rating  Dimension 

Means 

Standard 

Deviation 

Combined 
Peer/  Supervisor 
Reliabilities 

Selling  Skills 

26.04 

5.06 

.81 

Fluman  Relations  Skills 

13.29 

2.52 

.64 

Administrative  Skills 

6.10 

1.48 

.60 

Overall  Performance  Composite 

52.49 

8.81 

.80 

20 


Production  I  ndex  for  the  Concurrent  Validation  Study 


As  with  the  predictive  validation  sample,  mean  monthly  number  of  recruit  contracts 
signed  served  as  the  production  criterion.  Production  data  were  drawn  from  CNRC  for 
the  time  period  January  2000  through  December  2002.  Descriptive  statistics  for 
production  in  CNRC  and  for  the  concurrent  validation  sample  are  presented  in  Table  19. 


Table  19 

Average  monthly  uncorrected  (raw)  production:  Descriptive  statistics 


N 

Minimum 

Maximum 

Mean 

Standard 

Deviation 

Validation  Sample 

254 

0.11 

2.80 

1.17 

0.48 

All  of  CNRC 

7523 

0.03 

4.63 

1.02 

0.60 

As  we  did  previously,  reliabilities  were  computed  for  2-12  months  data,  and  these 
results  are  depicted  in  Table  20.  Again,  it  seemed  reasonable  to  screen  out  recruiters  in 
the  sample  with  fewer  than  four  months  of  production  data.  This  resulted  in  14 
additional  recruiters  dropped  from  the  sample. 


Table  20 

Reliabilities  of  production  indices  using  different  time  intervals 


Time  Length 

Reliability 

12  months 

.86 

11  months 

.86 

10  months 

.85 

9  months 

.85 

8  months 

.83 

7  months 

.81 

6  months 

.80 

5  months 

.75 

4  months 

.72 

3  months 

.66 

2  months 

.61 
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Correlations  between  the  Criterion  Measures 

Table  21  depicts  the  relationships  between  the  three  performance  rating  factors  and 
the  Region-adjusted  production  index.  Production  again  correlates  highest  with  the 
Selling  Skills  rating  factor  (.52),  the  factor  that  most  closely  represents  the  skill,  effort, 
and  ability  of  recruiters  to  enlist  prospects.  Production  correlates  less  highly  with  the 
Human  Relations  Skills  (.28)  and  Organizing  Skills  (.10)  rating  factors.  As  in  the 
predictive  validation  sample,  this  pattern  of  relations  provides  construct  validity 
evidence  for  both  the  ratings  and  production  index. 


Table  21 

Correlations  between  criterion  measures 


Criterion  Measure 

Production 

Selling 

Skills 

Human 

Relations 

Skills 

Organizing 

Skills 

Overall 

Performance 

Selling  Skills 

.52** 

Human  Relations  Skills 

28** 

.74** 

Organizing  Skills 

.10 

44** 

4Q*  * 

-na- 

N=  197 

*  <  .05 

*  p  <  .01 


Predictor  Data 

The  RAB  was  administered  to  254  recruiters  in  the  3  NRDs  that  participated  in  the 
concurrent  validation  study.  In  order  to  detect  random  or  careless  responding  by 
participants,  the  data  were  screened  using  two  validity  check  items,  identical  to  what 
was  used  in  the  WSI,  and  by  visual  inspection  of  the  answer  sheets.  A  total  of  30  tests 
(11.8%)  was  flagged  for  invalid  responses  and  were  subsequently  removed  from  the 
database.  Descriptive  statistics  for  each  of  the  RAB  factors  are  displayed  in  Table  22  and 
23- 


Table  22 

RAB  descriptive  statistics 


Scale 

N 

Mean 

Standard 

Deviation 

Ambition 

224 

40.05 

5.64 

Conscientiousness 

224 

57.25 

8.84 

Extroversion 

224 

99.69 

16.80 

Leadership 

224 

19.28 

3.98 

Emotional  Stability 

224 

50.87 

5.84 
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Concurrent  Validation  Results 

For  this  study,  similar  to  the  results  of  the  predictive  validation  study,  Extroversion 
was  significantly  correlated  with  Selling  Skills  (.21),  Human  Relations  Skills  (.20), 
Overall  Performance  (.20),  and  Production  (.16).  Ambition  was  valid  against  Selling 
Skills  (.22),  Human  Relations  Skills  (.14),  and  Overall  Performance  (.20). 
Conscientiousness  was  only  valid  against  Organizing  Skills  (.18)  and  Emotional  Stability 
was  only  valid  against  production  (.19).  The  Leadership  factor  was  not  significantly 
related  to  any  of  the  criteria  (see  Table  23). 


Table  23 

Correlations  between  RAB  factors  and  criteria 


Selling 

Skills 

Human 

Relations 

Skills 

Organizing 

Skills 

Overall 

Performance 

Production 

Ambition 

22** 

.14* 

-.01 

20** 

.13 

Conscientiousness 

.03 

.04 

* 

OO 
1— 1 

.06 

.01 

Extroversion 

2i** 

20** 

-.08 

20** 

.16* 

Leadership 

.09 

.10 

-.02 

.09 

.12 

Emotional  Stability 

.13 

.12 

-.02 

.12 

.19** 

*p<.05,  **p<.01 
N  =  199  for  Rating  Criteria 
N  =  200  for  Production 


Development  of  I  nterim  Final  Scoring  Key 


To  develop  a  scoring  key  based  on  the  validation  research  accomplished  here  and  to 
estimate  the  likely  validity  of  that  key,  we  completed  the  following  additional  steps:  (1) 
used  the  item-level  concurrent  validities  to  select  items  for  the  scoring  key;  (2)  formed  a 
composite  of  those  items;  (3)  cross-validated  the  composite  in  the  predictive  validation 
sample;  and  (4)  developed  an  interim  final  key  by  including  primarily  items  with  good 
validities  in  the  predictive  sample.  The  rationale  for  this  approach  was,  first  that  the  best 
estimates  of  the  likely  operational  validities  come  from  the  predictive  study.  In  that 
study,  the  predictors  were  administered  before  the  recruiter  trainees  had  any  recruiting 
experience,  similar  to  what  would  be  the  case  in  their  operational  use. 

Accordingly,  we  wanted  to  establish  a  “placeholder”  key  for  the  concurrent  study,  but 
then  evaluate  the  cross-validity  of  that  key  using  the  predictive  sample.  This  should 
provide  a  relatively  conservative  estimate  of  the  RAB’s  overall  validity. 

The  second  part  of  the  rationale  for  this  approach  is  that  although  the  Extroversion 
and  Emotional  Stability  scales  had  reasonably  consistent  validities  across  the  two 
studies,  arguing  perhaps  for  using  a  composite  of  those  two  scales  as  the  key,  there  were 
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several  of  those  scales’  items  that  showed  poor  validities  and  several  Ambition  and 
Leadership  items  that  had  very  good  validities.  Thus,  a  combination  of  the  reasonably 
large  sample  sizes  providing  good  estimates  of  item  level  validities  and  the  fact  that 
several  items  from  two  of  the  less  valid  scales  had  promising  validities  argued  for  this 
strategy. 

Finally,  the  third  part  of  the  rationale  is  that  the  items  for  the  interim  final  key  going 
forward  were  mostly  items  from  the  placeholder  key  used  to  provide  validity  estimates 
for  the  entire  RAB.  However,  a  few  of  the  placeholder  key  items  were  dropped  if  they 
had  very  poor  cross-validities  in  the  predictive  sample.  Also,  a  few  items  were  added  to 
the  key  if  they  were  not  part  of  the  placeholder  key  derived  from  the  concurrent  study, 
but  had  very  good  validity  in  the  predictive  sample,  especially  against  production,  Sales 
Skills,  and  Overall  Performance. 

Validation  results  for  the  item-level  analyses  appear  in  Tables  24  and  25.  As 
expected,  validities  for  the  concurrent  sample,  where  the  items  were  selected  based 
especially  on  their  correlations  with  production,  Selling  Skills,  and  Overall  Performance, 
are  as  high  as  .28.  However,  cross-validation  results  in  the  predictive  sample  are  also 
promising,  reaching  .21  for  production  and  .26  and  .27,  respectively,  against  the  Sales 
Skills  and  Overall  Performance  rating  criteria. 


Table  24 

RAB  score  descriptive  statistics 


Scale 

N 

Mean 

Standard  Deviation 

RAB  Score  (in  CV) 

224 

122.72 

16.38 

RAB  Score  (in  PV) 

537 

138.33 

13.16 

Table  25 

Correlations  between  RAB  score  and  criteria 


Selling 

Skills 

Human 

Relations 

Skills 

Organizing 

Skills 

Overall 

Performance 

Production 

RAB  Score  (in  CV) 

28** 

.21** 

-.02 

.25** 

28** 

RAB  Score  (in  PV) 

26** 

26** 

-.03 

.27** 

.21** 

*  p  <  .05 

**  p  <  .01 

for  CV  N  =  200  for  production;  199  for  ratings 
for  PV  N  =  537  for  production;  127  for  ratings 
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As  mentioned,  the  final  step  in  identifying  the  RAB  items  for  the  key  was  to  drop 
from  the  placeholder  key  (the  key  based  on  the  concurrent  validation  results)  items  that 
failed  to  cross-validate  in  the  predictive  sample  and  add  items  that  were  not  part  of  the 
placeholder  key  if  they  were  close  to  qualifying  for  the  key  and  showed  high  validities  in 
the  predictive  sample.  This  interim  final  key  contains  51  items  from  the  Extroversion, 
Emotional  Stability,  Ambition,  Leadership,  and  Conscientiousness  scales. 


Recommendations  for  Operational  Use 


Although  we  have  demonstrated  reasonably  good  validity  for  the  RAB,  there  is 
certainly  a  possibility  that  Sailors  taking  the  test  in  an  operational  setting  where  the 
scores  will  be  used  to  select  or  screen  out  individuals  for  recruiting  duty  could  slant  their 
response  (i.e.,  fake  good  or  fake  bad).  Warnings  about  faking  and  persuasive  messages 
encouraging  honest  responses  may  help  (e.g.,  Hough  &  Furnham,  2003),  but  test  takers 
determined  to  slant  their  responses  may  still  be  able  to  do  so. 

One  strategy  that  would  likely  address  this  problem  is  to  administer  the  RAB 
routinely  to  all  Petty  Officers  soon  after  their  first  reenlistment.  Test  scores  could  then 
be  placed  in  a  file  and  later  used  to  encourage  or  discourage  coming  into  recruiting.  This 
strategy  might  also  be  expanded  to  include  other  special  assignment  ratings  such  as  drill 
instructor.  The  RAB  could  then  be  used  as  more  of  a  classification  or  career  counseling 
tool  than  a  selection  tool. 
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Appendix  A: 

Navy  Recruiter  Performance  Rating  Scales 


A-o 


A-i 


Navy  Recruiter  Performance  Rating  Scales 


A- 2 


A.  Locating  And  Contacting  Qualified  Prospects 

Contacting  large  numbers  of  persons  likely  to  join  the  Navy;  skillfully  using  available  recruiting  aids  to  get  the  attention  of  young 
persons  eligible  for  Navy  service;  knowing  where  and  when  to  prospect;  persisting  in  prospecting  and  following  up  on  leads  even 


under  considerable  adversity. 
Exerts  little  effort  prospecting; 

Exerts  effort  prospecting,  but 

Uses  a  number  of  sources  and 

Displays  exceptional  ingenuity 

for  example,  often  fails  to 

may  use  a  limited  number  of 

recruiting  tools  for  prospecting 

and  energy  and  uses  a  wide 

follow  up  on  even  promising 

recruiting  tools  and  may  not 

and  is  effective  at  locating  and 

variety  of  recruiting  tools  very 

leads,  and  uses  recruiting  tools 

spend  enough  time  with  or 

contacting  qualified  prospects. 

effectively  to  locate  and 

(e.g.,  telephone,  RTools) 

direct  sufficient  effort  toward 

contact  qualified  prospects. 

sparingly  and  ineffectively. 

the  most  productive  sources 

and  prospects. 

*  * 

*  *  # 

-$■  ft 

<3>  © 

Needs  Considerable 
Improvement 

Needs  Improvement 

Effective 

Very  Effective 

B.  Gaining  And  Maintaining  Rapport _ 

Being  hospitable  to  prospects  in  the  office;  gaining  the  trust  and  respect  of  prospects;  adjusting  to  applicants'  styles  and  acting 
appropriately  with  different  types  of  applicants. 


Is  very  poor  at  gaining  and 
maintaining  rapport;  may 
display  a  lack  of  interest  to 
individual  prospects  or  may 
answer  questions  in  an 
impersonal  way. 

*  * 

Needs  Considerable 
Improvement 


Has  trouble  interacting  with 
certain  prospects;  sometimes 
appears  disinterested  in  a 
prospect  or  may  have  a 
standard  approach  to 
interacting  that  is  inappropriate 
for  some  prospects. 

#  * 

Needs  Improvement 


Is  typically  able  to  put 
prospects  at  ease,  maintains 
good  rapport  with  them,  and 
interacts  with  most  prospects 
in  a  warm  and  friendly  way. 

4-  -v-  n 

Effective 


Interacts  very  effectively  with 
all  types  of  prospects;  is 
excellent  at  gaining  and 
maintaining  rapport  and 
establishing  trust  with 
prospects. 

<3>  © 

Very  Effective 


C.  Obtaining  Information  From  Prospects  And  Making  Good  Person-Navy  Fits 

Making  accurate  judgments  about  prospects'  preferences  and  needs,  based  on  good  interviewing  skills;  effectively  obtaining 
information  about  prospects  from  other  sources  (e.g.,  high  school  principal,  parents)  to  assess  their  qualifications  and  needs. 

Is  very  poor  at  getting 

Sometimes  fails  to  learn 

Is  good  at  blueprinting  most 

Always  blueprints  effectively, 

prospects  to  reveal  their  needs 

enough  about  prospects  to 

prospects,  evaluating  their 

identifying  prospects’  needs 

and  buying  motives,  making  it 

identify  their  primary  needs 

needs  and  then  discussing 

and  career  motivations  and 

highly  difficult  to  suggest 

and  buying  motives;  may 

Navy  opportunities  appropriate 

then  is  excellent  at 

appropriate  Navy 

suggest  Navy  features  and 

for  meeting  those  needs. 

emphasizing  Navy  features  and 

opportunities;  may  suggest 

benefits  that  do  not  result  in  a 

opportunities  that  address  these 

> 

features  or  programs  that 

good  match  with  the 

needs/ motivations . 

4^ 

clearly  don’t  interest  prospect. 

individual’s  needs. 

*  * 

#  *  # 

4-  ft 

<3>  © 

Needs  Considerable 
Improvement 

Needs  Improvement 

Effective 

Very  Effective 

D.  Salesmanship  Skills _ 

Skillfully  persuading  prospects  to  join  the  Navy;  using  Navy  benefits  and  opportunities  effectively  to  sell  the  Navy;  adapting  selling 
techniques  appropriately  to  different  prospects,  effectively  overcoming  objections  to  joining  the  Navy. 


Fails  to  describe  Navy 
features/benefits  important  to 
individual  prospects,  and  is 
frequently  unable  to  counter 
objections  to  joining  the  Navy; 
often  misses  opportunities  to 
close  even  when  it’s  clearly 
appropriate  to  do  so. 

*  * 

Needs  Considerable 
Improvement 


Describes  Navy  features  and 
benefits  in  a  way  that  is 
sometimes  not  suited  to  an 
individual  prospect’s  interests 
or  needs  and  may  not 
recognize  prospect’s  criticisms 
or  objections  to  the  Navy;  at 
times,  misses  opportunities  to 
close. 

#  * 

Needs  Improvement 


Describes  Navy 
features/benefits  so  that  most 
prospects  become  more 
interested  in  the  Navy  and  is 
prepared  to  counter  frequently 
heard  objections  about  the 
Navy;  knows  when  and  how  to 
close  in  most  situations. 

4-  n 

Effective 


Describes  Navy  life  and 
benefits  in  a  highly  appropriate 
and  convincing  way  for  each 
prospect,  and  is  very  adept  at 
answering  questions  about  the 
Navy  or  countering  any 
objections;  never  misses 
opportunities  to  close. 

<3>  O 

Very  Effective 
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E.  Establishing  And  Maintaining  Good  Relationships  In  The  Community _ 

Contacting  and  working  effectively  with  individuals  and  agencies  capable  of  helping  with  prospects;  presenting  a  good  image  and 
building  a  good  reputation  for  the  Navy  by  developing  positive  relationships  with  persons  in  the  community. 


Alienates  individuals  in  the 
community  by  failing  to  honor 
commitments,  making 
demands,  or  avoiding  personal 
contact  altogether;  presents 
negative  image  of  the  Navy  by 
poor  personal  appearance  or 
behavior. 

*  * 

Needs  Considerable 
Improvement 


Does  not  make  regular  contact 
with  community  agencies  that 
might  be  helpful  in  recruiting, 
and  fails  to  develop 
relationships  fully;  is  not 
particularly  alert  to 
opportunities  to  promote  the 
Navy  in  the  community. 

#  *  # 

Needs  Improvement 


Spends  productive  time  with 
individuals/agencies  in  the 
community,  and  keeps  them 
informed  about  most  Navy 
activities;  may  arrange  Navy 
activities  for  community 
persons  who  can  help  in 
recruiting. 

Effective 


Is  exceptionally  adept  at 
developing  excellent 
relationships  with  relevant 
individuals  and  community 
agencies,  and  projects  a  very 
positive  image  of  the  Navy; 
may  volunteer  off-duty  time  to 
help  in  the  community. 

<3>  © 

Very  Effective 


F.  Providing  Knowledgeable  And  Accurate  Information  About  The  Navy _ 

Displaying  considerable  knowledge  about  Navy  opportunities,  and  answering  questions  in  a  competent  manner;  providing  accurate 
information  about  Navy  life,  so  that  the  prospect  or  recruit  is  fully  informed  but  not  discouraged  from  joining  the  Navy;  being  up  to 
date  on  changes  in  the  Recruiting  Manual  and  Navy  directives.  _ _ 


Frequently  fails  to  provide 
accurate  information  about 
Navy  opportunities  or  benefits, 
and  often  fails  to  prepare 
recruits  for  Navy  life  or  RTC; 
spends  little  time  learning 
about  Navy  opportunities. 

*  * 

Needs  Considerable 
Improvement 


Provides  infonnation  about 
certain  aspects  of  Navy  life 
and  opportunities,  but  some  of 
it  may  be  inaccurate  or 
incomplete;  is  not 
knowledgeable  about  many 
features/benefits  of  the  Navy. 

#  *  # 

Needs  Improvement 


Keeps  current  on  most  Navy 
opportunities,  and  competently 
answers  questions  from 
prospects;  prepares  applicants 
well  for  the  recruitment 
process. 

4-  n 

Effective 


Consistently  provides  detailed 
and  accurate  information  about 
Navy  life,  available 
opportunities,  RTC,  etc.; 
always  keeps  up-to-date  on 
new  Navy  directives  and 
policies  relevant  for  recruiting. 

O 

Very  Effective 
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G.  Administrative  Skills _ 

Planning  ahead;  organizing  time  efficiently;  completing  paperwork  accurately  and  on  time;  keeping  track  of  appointments;  not 
wasting  time. 


Consistently  fails  to  complete 
necessary  forms  or  may  use 
wrong  forms;  organizes  time 
very  poorly  and  does  not 
maintain  applicant  log/planner. 


*  * 

Needs  Considerable 
Improvement 


Sometimes  completes 
paperwork  late,  occasionally 
with  significant  errors;  is 
somewhat  inefficient  in  use  of 
time,  and  may  at  times 
schedule  appointments  without 
considering  other  events. 

*  *  # 

Needs  Improvement 


Usually  completes  paperwork 
on  time  and  with  few  errors; 
keeps  a  complete  and  accurate 
applicant/log  planner,  and 
generally  uses  time  efficiently. 


4-  ft 

Effective 


Finishes  all  paperwork 
accurately  and  on  time;  devises 
plans  so  as  to  achieve  own  and 
district  goals;  maintains 
complete  calendar  of  relevant 
events  and  schedules  work 
activities  very  efficiently  and 
effectively. 

<3>  © 

Very  Effective 


H.  Supporting  Other  Recruiters  And  The  Command _ 

Coordinating  activities  with  other  recruiters  to  maximize  the  productivity  of  the  station  and  district;  using  own  time  to  support  other 
Navy  recruiters  when  appropriate;  providing  constructive  feedback  and  helpful  tips  to  other  recruiters,  especially  if  they  are  new. 


Is  a  very  poor  team  player, 
placing  own  goals  ahead  of  the 
group  and  often  refusing  to 
help  other  recruiters;  is  quick 
to  blame  others  when  the 
station  does  not  achieve  goal; 
lets  others  carry  the  recruiting 
load. 

*  * 

Needs  Considerable 
Improvement 


Helps  and  provides  feedback 
to  other  recruiters,  but  only 
when  asked;  may  complain 
about  production  quotas,  or 
having  to  work  extra  time. 


*  *  -$■ 
Needs  Improvement 


Supports  the  Command  in 
ways  that  are  helpful; 
generally  works  with  other 
recruiters  to  help  them  or  to 
improve  their  skills. 


4-  J3 

Effective 


Is  always  enthusiastic  and 
works  to  build  group  spirit; 
consistently  helps  other 
recruiters,  even  when  he/she  is 
busy;  always  shares 
information  so  as  to  increase 
group  production. 

<3>  © 

Very  Effective 


Appendix  B: 

Navy  Recruiter  Participation  I  nformation  Sheet 


B-o 


Participant  Information  Sheet 

Note:  Please  print  your  responses.  All  in  formation  is  strictly  con  fidential. 

1.  Name _ 

Last  First  MI  Rank 

2.  SSN _ 

3.  Age _  4.  Gender  Male _  Female _ 

5.  How  long  have  you  been  in  the  Navy? _ Years _ Months 

6.  Where  is  your  first  recruiting  assignment? _ 


7.  What  is  your  current  marital  status  (place  a  check  before  the  appropriate  category)? 

a.  _ Married  c. _ Engaged  e. _ Living  together 

b.  _ Divorced/Widowed/Separated  d. _ Never  been  married 

8.  Which  ethnic  group  best  describes  you? 

a.  _ White/Caucasian  d. _ Hispanic/Latino/Mexican  American 

b.  _ African  American/Black  e. _ Native  American/Alaskan  Native 

c.  _ Asian/Pacific  Islander  f. _ Other  (please  specify) 


9.  What  is  the  highest  level  of  education  you  have  completed? 

a.  _ Less  than  High  School  c. _ Some  College  e. _ College 

Graduate 

b.  _ High  School  Graduate/GED  d. _ Associate’s  Degree 

10.  Please  rate  each  of  the  statements  listed  below  using  the  following  5 -point  scale. 

1  strongly  disagree 

2  somewhat  disagree 

3  neither  agree  nor  disagree 

4  somewhat  agree 

5  strongly  agree 

_  I  would  be  very  happy  to  spend  the  rest  of  my  career  with  the  Navy 

_  I  owe  a  great  deal  to  the  Navy 
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I  feel  I  have  too  few  options  to  consider  leaving  the  Navy 
I  do  not  feel  a  strong  “sense  of  belonging”  to  the  Navy 

Too  much  of  my  life  would  be  disrupted  if  I  decided  I  wanted  to  leave  the  Navy  now 
I  do  not  feel  any  obligation  to  remain  with  the  Navy 


B-2 
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Navy  Recruiter  Predictor  I  nstructions  Script 

First,  introduce  yourself  and  then  say: 

We  have  been  asked  by  CNRC  to  develop  and  evaluate  new  survey  instruments  that 
can  be  used  to  identify  individuals  with  high  potential  for  recruiting  duty.  To  accomplish 
this  we  will  be  administering  these  experimental  surveys  to  all  incoming  ENRO  students 
over  the  next  six  months  and  then  following  up  to  evaluate  the  ability  of  the  surveys  to 
identify  effective  recruiters.  Let  me  emphasize  that  the  surveys  are  experimental  at  this 
point,  so  there  are  no  right  or  wrong  answers.  Also,  your  individual  responses  will  not  be 
reported  to  the  Navy  and  will  not  affect  your  career  in  any  way.  We  are  not  evaluating 
you;  we’re  evaluating  the  survey  instruments’  usefulness  as  a  screening  tool  for  NCOs 
being  considered  for  recruiting  duty  sometime  in  the  future.  Please  answer  the  survey 
items  honestly  and  accurately. 

OK,  let’s  go  over  some  operational  details.  We  have  tried  to  make  this  process  as  easy 
as  possible  so  please  bear  with  us  as  we  go  through  these  procedures. 

Operational  I  nstructions 

1.  Stapled  to  the  front  cover  of  your  packet  is  an  Instruction  Sheet  for  each  of  the 
four  instruments  we  are  asking  you  to  complete  today.  It  is  important  that  you 
complete  ALL  FOUR  instruments  and  their  appropriate  answer  sheets  in  the 
order  specified  on  the  Instruction  Sheet.  This  instruction  sheet  has  important 
information  for  filling  out  each  answer  sheet.  For  example,  as  you  can  see  on  the 
Instruction  Sheet,  for  the  first  instrument  called  the  “Special  Assignment  Battery 
Work  Style  Inventory”  we  ask  you  to  put  your  social  security  number  in  the 
spaces  marked  ID  Number  on  the  answer  sheet.  There  will  be  separate 
instructions  for  each  of  the  four  instruments.  However,  for  all  instruments,  we 
ask  that  you  DO  NOT  WRITE  IN  ANY  OF  THE  TEST  BOOKLETS  and  that  you 
are  sure  to  put  your  social  security  number  on  every  answer  sheet.  Again  your 
responses  on  all  of  the  items  will  be  kept  confidential. 

2.  Note:  If  they  ask  why  they  need  to  put  their  social  security  numbers  and  names 
on  the  forms,  explain  to  them  that  the  information  from  these  tests  will  be 
analyzed  with  their  performance  data  at  a  later  date.  Assure  them  that  the 
results  of  these  analyses  will  only  be  reported  on  a  group  level  to  the  Navy. 

3.  Now  open  your  packets.  The  first  page  in  your  packet  should  be  the  Participant 
Information  Sheet.  ( Hold  one  up).  Please  take  a  few  minutes  now  and  complete 
this  information  sheet.  (Wait  for  them  to  finish). 
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4-  You  should  also  have  a  total  of  four  instruments  and  answer  sheets.  The  first 
instrument  is  the  “Special  Assignment  Battery  Work  Style  Inventory”  and  has  a 
blue  answer  sheet  marked  General  Purpose  -  NCS-  Answer  Sheet  ( Hold  one  up). 
The  second  instrument  is  called  the  “Vocational  Interest  Career  Examination  for 
Navy  Recruiters”  and  has  an  orange  answer  sheet  ( Hold  one  up),  the  third  is  the 
NEO  PI-R  with  a  navy  blue  answer  sheet  ( Hold  one  up)  and  the  fourth  is  a  purple 
EQ-i  booklet  with  a  black  and  white  answer  sheet  ( Hold  one  up).  If  you  are 
missing  any  of  these  booklets  or  answer  sheets,  please  raise  your  hand  and  I  will 
give  you  what  you  are  missing. 

5.  Please  let  me  know  if  you  have  any  questions  before  we  begin  (Pause).  Thank  you 
very  much  for  your  help  with  this  important  project. 
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RAB  Scale  Definitions 
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RAB  Scale  Definitions 


Ambition 

The  tendency  to  set  high  work  standards  and  strive  to  meet  them.  Persons  who  score 
high  on  this  scale  are  more  willing  to  make  sacrifices  to  meet  their  goals,  are  optimistic 
about  their  future,  and  believe  that  they  will  be  successful. 


Extroversion 

The  tendency  to  be  engaging,  lively,  and  sociable.  Persons  who  score  high  on  this 
scale  are  energetic,  approachable,  have  lots  of  friends,  and  enjoy  being  the  center  of 
attention. 


Emotional  Stability 

The  tendency  to  have  a  happy,  positive  attitude  toward  life.  Persons  who  score  high 
on  this  scale  are  more  self  confident  and  content,  whereas  persons  who  score  low  on  this 
scale  need  more  reassurance,  see  life  as  more  unfair,  and  have  difficulty  coping  with 
stress. 

Leadership 

The  tendency  to  enjoy  having  power  or  influence  over  others  and  to  prefer  leadership 
positions. 


Conscientiousness 

The  tendency  to  be  planful,  organized,  and  rule-abiding.  Persons  who  score  high  on 
this  scale  are  more  deliberate  and  orderly  in  their  activities,  whereas  persons  who  score 
low  are  more  carefree,  easygoing,  and  spontaneous. 
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Participant  I  nformation  Sheet  for  the  Concurrent 

Validation  Study 
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Participant  Information  Sheet 

Note:  Please  print  your  responses.  All  information  is  strictly  con  fidential. 

1.  Name _ 

Last  First  MI  Rank 


2.  Age _  3.  Gender  Male  _  Female _ 

4.  How  long  have  you  been  in  the  Navy? _ Years _ Months 

5.  What  is  your  current  job/position?  □  Recruiter  □  RINC  □  Zone  Supervisor 

□  Other _ 

6.  How  long  have  you  been  in  your  current  job/position? _ Years _ Months 

7.  Current  Zone  location:  _ 

8.  Current  Station  location  (if  applicable): _ 

9.  What  is  your  current  marital  status  (place  a  check  before  the  appropriate  category)? 

a.  _ Married  c. _ Engaged  e. _ Living  together 

b.  _ Divorced/Widowed/Separated  d. _ Never  been  married 

10.  Which  ethnic  group  best  describes  you? 

a.  _ White/Caucasian  d. _ Hispanic/Latino/Mexican  American 

b.  _ African  American/Black  e. _ Native  American/Alaskan  Native 

c.  _ Asian/Pacific  Islander  f. _ Other  (please  specify) 


11.  What  is  the  highest  level  of  education  you  have  completed? 
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c. _ Some  College 


e. _ College 


a.  _ Less  than  High  School 

Graduate 

b.  _ High  School  Graduate/GED  d. 


Associate’s  Degree 
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I  nformed  Consent  Form  for  the  Concurrent  Validation 

Study 
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INFORMED  CONSENT  AND  PRIVACY  STATEMENT 


The  overall  objective  of  this  project  is  to  evaluate  new  recruiter  selection  tools.  The  end  goal  is 
to  produce  a  selection  tool  that  will  ensure  that  sailors  with  a  high  potential  for  success  in 
recruiting  duty  will  be  selected  and  trained  to  be  recruiters  in  order  to  maximize  overall  recruiter 
productivity.  To  accomplish  this,  we  need  you  to  provide  performance  ratings  of  one  (or  possibly 
more)  of  the  recruiters  working  for  or  with  you.  These  performance  ratings  will  be  confidential, 
for-research-only  evaluations.  A  coding  system  will  be  used  that  ensures  no  names  will  appear 
on  the  actual  rating  form.  Researchers  will  do  all  of  the  data  analyses,  and  the  Navy  will  not  see 
any  individual  level  data. 

Authority  to  request  this  infonnation  is  granted  under  10  U.S.C.  5031  and  5032,  and  5  U.S.C. 
301:  Executive  Order  9397.  Your  answers  are  confidential.  No  one  will  be  able  to  identify  you 
personally. 

PURPOSE:  The  purpose  of  this  study  is  to  identify  measures  for  a  selection  protocol  that  can  be 
developed  and  provided  to  sailors  prior  to  being  assigned  to  recruiting  duty.  This  protocol  will 
increase  the  likelihood  of  improved  recruiter  performance  and  productivity. 

ROUTINE  USES:  The  information  provided  in  this  questionnaire  will  be  analyzed  by  Personnel 
Decisions  Research  Institutes,  Inc.  (PDRI)  and  Navy  Personnel  Research,  Studies,  and 
Technology  (NPRST). 

CONFIDENTIALITY :  All  responses  will  be  held  in  confidence  by  NPRST  and  PDRI. 
Information  you  provide  will  be  statistically  summarized  with  the  responses  of  other  recruiters 
and  will  not  be  attributable  to  any  single  individual. 

PARTICIPATION :  Participation  in  this  study  is  entirely  voluntary.  Failure  to  respond  to  any  of 
the  questions  will  not  result  in  any  penalties  to  you.  However,  your  views  are  extremely 
important  and  will  be  critical  for  completion  of  the  recruiter  selection  tool  and  thus  ensuring  the 
selection  of  individuals  with  the  highest  potential  for  success  in  recruiting  duty. 

STATEMENT  OF  RISK:  The  data  collection  procedures  are  not  expected  to  involve  any  risk  or 
discomfort  to  you. 

If  you  have  any  questions  about  this  questionnaire,  please  contact  Mr.  Ron  Bearden,  DSN  882-2972  or  COM 
901-874-2972.  For  questions  regarding  Human  Subjects  issues  contact  NPRST  Protection  of  Human  Subjects 
Committee,  DSN  882-3086  or  COM  901-874-3086  or  e-mail  IRB@persnet.navy.mil. 

I  have  read  the  Informed  Consent  and  Privacy  Statements  and  wish  to  proceed  with  the 
questionnaire. 


Signature 


Date 
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«Rater  Code» 


Performance  Rating  Sheet  Rater  Code  Number: 


Mark  your  answers  by  blackening  the  appropriate  circle 


Ratee  Code  Number: 

«Rateel_Code» 

«Ratee2_Code» 

«Ratee3_Code» 

Your  relationship  to  the  ratee: 

Peer 

Peer 

Peer 

Supervisor 

Supervisor 

Supervisor 

Length  of  time  you’ve  worked  with  the  ratee: 

Years 

Years 

Years 

Months 

Months 

Months 

A.  Locating  And  Contacting  Qualified  Prospects 

©®®©©©@®®® 

©®®@©©®®®® 

©®®©©©@®®® 

B.  Gaining  and  Maintaining  Rapport 

©®®©©©®®®® 

©®®©©©@®®® 

©®®@©©®®®® 

C.  Obtaining  Information  From  Prospects  and  Making  Good 

©®®@©©®®®® 

©®®©©©@®®® 

©®®©©©®®®® 

Person-Navy  Fits 

D.  Salesmanship  Skills 

©®®@©©®®®® 

©®®@©©®®®® 

©®®@©©®®®® 

E.  Establishing  And  Maintaining  Good  Relationships  In  The 

©®®©©©@®®® 

©®®©©©@®®® 

©®®©©©®®®® 

Community 

F.  Providing  Knowledgeable  And  Accurate  Information 

©®®@©©®®®® 

©®®@©©®®®® 

©®®©©©®®®® 

About  The  Navy 

G.  Administrative  Skills 

©®®@©©®®®® 

©®®©©©@®®® 

©®®©©©@®®® 

H.  Supporting  Other  Recruiters  And  The  Command 

©®®@©©®®®® 

©®®©©©@®®® 

©®®©©©®®®® 

Overall  Effectiveness 

©®®@©©®®®® 

©®®©©©@®®® 

©®®©©©®®®® 

Ratee  name: 

«Rateel_Name» 

«Ratee2_Name» 

«Ratee3_Name» 

Distribution 


AIR  UNIVERSITY  LIBRARY 

ARMY  MANAGEMENT  STAFF  COLLEGE  LIBRARY 
ARMY  RESEARCH  INSTITUTE  LIBRARY 
ARMY  WAR  COLLEGE  LIBRARY 
CENTER  FOR  NAVAL  ANALYSES  LIBRARY 
COMMANDER,  NAVY  RECRUITING  COMMAND  (N5) 

DEFENSE  TECHNICAL  INFORMATION  CENTER 
HUMAN  RESOURCES  DIRECTORATE  TECHNICAL  LIBRARY 
JOINT  FORCES  STAFF  COLLEGE  LIBRARY 
MARINE  CORPS  UNIVERSITY  LIBRARIES 
NATIONAL  DEFENSE  UNIVERSITY  LIBRARY 

NAVAL  HEALTH  RESEARCH  CENTER  WILKINS  BIOMEDICAL  LIBRARY 
NAVAL  POSTGRADUATE  SCHOOL  DUDLEY  KNOX  LIBRARY 
NAVAL  RESEARCH  LABORATORY  RUTH  HOOKER  RESEARCH  LIBRARY 
NAVAL  WAR  COLLEGE  LIBRARY 

NAVY  PERSONNEL  RESEARCH,  STUDIES,  AND  TECHNOLOGY  SPISHOCK 
LIBRARY  (3) 

PENTAGON  LIBRARY 

USAF  ACADEMY  LIBRARY 

US  COAST  GUARD  ACADEMY  LIBRARY 

US  MERCHANT  MARINE  ACADEMY  BLAND  LIBRARY 

US  MILITARY  ACADEMY  AT  WEST  POINT  LIBRARY 

US  NAVAL  ACADEMY  NIMITZ  LIBRARY 


